GPU accelerated encoder #895

dmanc · 2024-11-14T08:30:57Z

Why are these changes needed?

This PR adds support for multiple backends for the encoder. Initially we only have the gnark and icicle backends, see https://dev.ingonyama.com/icicle/overview for more information on the icicle backend.

A backend may include GPU acceleration as an option, see https://dev.ingonyama.com/icicle/install_cuda_backend for icicle's GPU accelerated backend.

In order to use the icicle backend it must be compiled with the icicle build tag:

go build -tags=icicle main.go

Checks

I've made sure the tests are passing. Note that there might be a few flaky tests, in that case, please comment that they are not relevant.
I've checked the new test coverage and the coverage percentage didn't drop.
Testing Strategy
- Unit tests
- Integration tests
- This PR is not tested :(

samlaf

Yikes lost all my comments while reviewing on github.dev.... friggin github. Some graphql internal error or something. So I'll just submit this first set of reviews in case. Don't really feel like reviewing again lol... another reason why small PRs are gud, big PRs bad.

api/clients/retrieval_client_test.go

core/test/core_test.go

disperser/cmd/encoder/flags/flags.go

disperser/cmd/encoder/config.go

samlaf

Just spent 2 hours reviewing and I barely even got to the important stuff. Suggest no more PRs like this, for the sake of our industry, and of your reviewers' mental health. :D

docker-bake.hcl

encoding/bench/Makefile

samlaf · 2024-11-14T11:47:01Z

encoding/icicle/device_setup.go

+type IcicleDevice struct {
+	Device         icicle_runtime.Device
+	NttCfg         core.NTTConfig[[icicle_bn254.SCALAR_LIMBS]uint32]
+	MsmCfg         core.MSMConfig
+	FlatFFTPointsT []icicle_bn254.Affine
+	SRSG1Icicle    []icicle_bn254.Affine
+}


Is the point that we can have a different ntt/msm config and points loaded on different devices (gpus?)? Do we actually make use of this flexibility? If so, please add comment explaining.

No there's only going to be a single configuration loaded at startup time. Right now we don't support multiple GPUs for an encoder.

Is this mentioned anywhere? Would be good to if not.

encoding/icicle/device_setup.go

samlaf · 2024-11-14T11:49:31Z

encoding/icicle/device_setup.go

+			setupErr = fmt.Errorf("could not setup NTT")
+			return


just return the setupErr directly?

also prob want to wrap the icicleErr to help debug

cannot return because I don't think that RunOnDevice func will let me. Looks like this:

func RunOnDevice(device *Device, funcToRun func(args ...any), args ...any) { go func(deviceToRunOn *Device) { defer runtime.UnlockOSThread() runtime.LockOSThread() originalDevice, _ := GetActiveDevice() SetDevice(deviceToRunOn) funcToRun(args...) SetDevice(originalDevice) }(device) }

ah my bad I completely missed the function signature.
I see so the code passed to RunOnDevice is run on the gpu...? Did you test that your shared pointer for setupErr is working then? This would mean the gpu is able to go write in the cpu's RAM. Does this only work on mac where cpu/gpu share memory? Or is this something that always works?

No, RunOnDevice locks the OS thread so it makes all of that code run on the same OS thread. It's because with go routines you can call SetDevice to set the GPU device and that can run on thread 1 but the go routine can end up switching to thread 2 in the middle which means that it will use the default device (CPU).

encoding/icicle/device_setup.go

encoding/icicle/msm_setup.go

encoding/icicle/ntt_setup.go

jianoaix

Can we have a diff-neutral PR just for refactoring? It seems there are code here which doesn't need deep review as they are just existing code moved around?

encoding/rs/serialization.go

encoding/rs/noicicle.go

encoding/rs/serialization.go

dmanc · 2024-11-15T01:45:16Z

Can we have a diff-neutral PR just for refactoring? It seems there are code here which doesn't need deep review as they are just existing code moved around?

Yes, if we're happy with this new configuration method then I'll make another PR with that and rebase this one

dmanc · 2024-11-15T05:15:36Z

disperser/cmd/encoder/icicle.Dockerfile

+RUN go build -tags=icicle -o ./bin/server ./cmd/encoder
+
+# Start a new stage for the base image
+FROM nvidia/cuda:12.2.2-base-ubuntu22.04


Image size comparison:

ghcr.io/layr-labs/eigenda/encoder latest 781c3866c5dd 6 hours ago 41MB
6129846e8150 6 hours ago 41MB
ghcr.io/layr-labs/eigenda/encoder-icicle latest f530fe9c250d 21 hours ago 760MB

what makes it so large? near 20x is a lot

Didn't look into but my guess right now would be the cuda base image which contains the cuda runtime.

disperser/cmd/encoder/main.go

dmanc · 2024-11-15T19:34:00Z

encoding/kzg/prover/noicicle.go

+
+func CreateIcicleBackendProver(p *Prover, params encoding.EncodingParams, fs *fft.FFTSettings, ks *kzg.KZGSettings) (*ParametrizedProver, error) {
+	// Not supported
+	return nil, fmt.Errorf("icicle backend called without icicle build tag")


The point of noicicle.go file is that the icicle backend should only be functional when the icicle build tag is used. This is so we don't need to include the icicle backend files in the traditional encoder backend, otherwise it may break the existing build.

dmanc · 2024-11-15T19:34:55Z

encoding/kzg/prover/proof_backend.go

+}
+
+// CommitmentDevice represents a backend capable of computing various KZG commitments.
+type KzgCommitmentsBackend interface {


Moved commitments to its own backend in case we would like to accelerate them in the future.

dmanc · 2024-11-15T19:35:41Z

encoding/kzg/prover/prover.go

@@ -331,3 +457,99 @@ func toUint64Array(chunkIndices []encoding.ChunkNumber) []uint64 {
 	}
 	return res
 }
+
+func (p *Prover) newProver(params encoding.EncodingParams) (*ParametrizedProver, error) {


this file should be reviewed carefully

dmanc · 2024-11-15T19:46:13Z

encoding/kzg/verifier/verifier.go

 )

+// VerifierOption defines a function that configures a Verifier
+type VerifierOption func(*Verifier) error


This can go in it's own PR, it's applying the same options pattern refactor to verifier so not super relevant to GPU encoder

It looks more verbose and is deviating from the convention, what're the main problems it solves here compared to passing in a config struct?

One argument I would give for this configuration pattern is that it provides a simpler constructor that is extensible and backwards compatible.

For example, with a config struct we would have to modify every client that calls the constructor when we add a new configuration value vs. this pattern you add a WithX function in the library that sets the field and does whatever validation it needs to. Then clients can optionally update to use that new added configuration.

with a config struct we would have to modify every client that calls the constructor when we add a new configuration value

Not sure about this, if that new field is optional, the caller doesn't need to do anything. If it's required, callers will have to make changes in either case.

My concerns are the boilplate and the scalabilty of keeping adding WithXX, we may have a long list of fields (like in some big structs).
Also it looks it still need to operate on config struct, if the validation runs across multiple fields (eg. invariants defined over multiple fields).

My concerns are the boilplate and the scalabilty of keeping adding WithXX, we may have a long list of fields (like in some big structs).

These seem like tractable problems though, here is an example in grpc library:
https://github.com/grpc/grpc-go/blob/66385b28b3fe71a4895f00d581ede0a344743a3f/dialoptions.go#L248

Think I agree with both of you. I've grown fond of the options pattern and like it best, but I still think consistency across a codebase is the most important thing, so kind of agree with jian about this.

No problem, I'll revert to the previous method

dmanc · 2024-11-15T19:49:48Z

encoding/rs/parametrized_encoder.go

+	"github.com/consensys/gnark-crypto/ecc/bn254/fr"
+)
+
+type ParametrizedEncoder struct {


Refactor to make the encoder parametrized based on FFT settings (similar to Prover). Don't think it's too valuable since max time is ~600ms for loading largest fft setting. Am thinking of reverting

dmanc · 2024-11-15T19:51:24Z

encoding/icicle/utils.go

@@ -0,0 +1,103 @@
+//go:build icicle
+
+package icicle


This folder is icicle backend related setup

dmanc · 2024-11-15T19:51:53Z

encoding/kzg/prover/icicle/ecntt.go

@@ -0,0 +1,46 @@
+//go:build icicle
+
+package icicle


This folder is icicle related computations for MultiProofs

dmanc · 2024-11-15T19:52:06Z

encoding/rs/icicle/extend_poly.go

+	"github.com/ingonyama-zk/icicle/v3/wrappers/golang/runtime"
+)
+
+type RsIcicleComputeDevice struct {


This is icicle related computations for RS Encode

dmanc · 2024-11-15T19:54:12Z

One thing I'm not too sure how to deal with is testing Icicle/ GPU encoder correctness. The only strategy I can think of is to compare output of default backend and icicle backend.

Also if we wanted it to be part of CI, then we need to acquire GPU based github action runners.

encoding/rs/icicle.go

jianoaix · 2024-11-15T22:19:42Z

encoding/rs/icicle.go

+
+func CreateIcicleBackendEncoder(e *Encoder, params encoding.EncodingParams, fs *fft.FFTSettings) (*ParametrizedEncoder, error) {
+	icicleDevice, err := icicle.NewIcicleDevice(icicle.IcicleDeviceConfig{
+		GPUEnable: e.Config.GPUEnable,


It looks Icicle will support also CPU now?

Yes but it's not as performant as the gnark backend, am trying to look into why

jianoaix · 2024-11-15T22:23:33Z

encoding/rs/icicle/extend_poly.go

+
+	var evals []fr.Element
+	g.NttCfg.BatchSize = int32(1)
+	runtime.RunOnDevice(&g.Device, func(args ...any) {


If the GPU is enabled, then this func will be executed on GPU by the runtime?

yes because the device variable will be set to the GPU device here

eigenda/encoding/icicle/device_setup.go

Line 88 in dfe247d

Device: device,

For performance, it looks the main data movement to/from GPU will be the coefficients and the resulting evaluations?

For RSEncode yes:

For multiproofs it is also the same (move coefficients, move out proofs) but in addition we need to also transfer SRS Table to compute the MSM. This can be done at SetupMSM at initialization but not sure how much speedup that is.

jianoaix · 2024-11-15T22:24:49Z

encoding/kzg/verifier/verifier.go

 )

+// VerifierOption defines a function that configures a Verifier
+type VerifierOption func(*Verifier) error


It looks more verbose and is deviating from the convention, what're the main problems it solves here compared to passing in a config struct?

jianoaix · 2024-11-15T22:28:29Z

encoding/rs/encoder.go

-		Fs:             fs,
-		verbose:        verbose,
-		NumRSWorker:    runtime.GOMAXPROCS(0),
+	return &ParametrizedEncoder{


So when you say "backend", it looks a software/library that computes RS encoding? I.e. it decoupled device (CPU/GPU) and backend (the software that executes compute on device)

Right now I consider backend as library that implements the cryptographic primitive like NTT, MSM. We have our own implementation of NTT borrowed from protolambda and MSM from gnark.

It could be plausible that there exists libraries that implement the higher level primitives like multiproofs, commitment, RS encode and we can try to make our code more of a frontend that calls those. But for now we focus on the core primitives that give the speedup factor.

encoding/rs/extend_poly.go

bxue-l2 · 2024-11-19T23:49:43Z

encoding/kzg/prover/commitments.go

+	Srs        *kzg.SRS
+	G2Trailing []bn254.G2Affine
+}
+


would be good to put a comment, describing e(ComputeCommitment, g2_shifted) = e(g1, ComputeLengthCommitment)

should have been
e(g1_shifted, ComputeLengthCommitment) = e(g1, ComputeLengthProof)
e(ComputeCommitment, g2) = e(g1, ComputeLengthCommitment)

I decided to do documentation update in a separate PR

For documentation, I think this one appears to be quite readable: https://github.com/crate-crypto/go-kzg-4844/blob/master/internal/kzg/kzg_prove.go, which we can take some inspiration from

bxue-l2 · 2024-11-19T23:50:50Z

disperser/cmd/apiserver/main.go

@@ -162,7 +162,8 @@ func RunDisperserServer(ctx *cli.Context) error {
 	bucketName := config.BlobstoreConfig.BucketName
 	logger.Info("Blob store", "bucket", bucketName)
 	if config.DisperserVersion == V2 {
-		prover, err := prover.NewProver(&config.EncodingConfig, true)
+		config.EncodingConfig.LoadG2Points = true
+		prover, err := prover.NewProver(&config.EncodingConfig, nil)


the first arg is kzgConfig, according to the func. Name issue?

I think it's acceptable

encoding/kzg/prover/icicle.go

bxue-l2 · 2024-11-20T00:20:15Z

encoding/kzg/prover/prover.go

+		return nil, nil, fmt.Errorf("failed to create SRS table: %w", err)
+	}
+
+	fftPoints, err := subTable.GetSubTables(params.NumChunks, params.ChunkLength)


I don't think we ever need fftPoints, chance for optimization

Will include this in followup optimizations

bxue-l2 · 2024-11-20T00:33:01Z

encoding/rs/parametrized_encoder.go

+	jobChan := make(chan JobRequest, numWorker)
+	results := make(chan error, numWorker)
+
+	for w := uint64(0); w < numWorker; w++ {


try workerpool?

Will include in followup. Wrote ticket

jianoaix

LG mostly, a bunch of small comments

jianoaix · 2024-11-20T18:51:04Z

disperser/cmd/encoder/icicle.Dockerfile

+RUN go build -tags=icicle -o ./bin/server ./cmd/encoder
+
+# Start a new stage for the base image
+FROM nvidia/cuda:12.2.2-base-ubuntu22.04


what makes it so large? near 20x is a lot

jianoaix · 2024-11-20T18:55:43Z

disperser/cmd/encoder/main.go

+	encodingConfig := &encoding.Config{
+		BackendType: backendType,
+		GPUEnable:   config.ServerConfig.GPUEnable,
+		NumWorker:   config.EncoderConfig.NumWorker,


Do we have understanding about how this parallelism interacts with the blob level parallelism at the server?

From my understanding the interaction is that it can introduce CPU contention if we are handling more than 1 blob concurrently. In the worst case we set NumWorkers = # CPU on machine and two blobs are doing computations that require all workers.

jianoaix · 2024-11-20T19:48:24Z

encoding/icicle/device_setup.go

+		}
+
+		// Setup MSM if parameters are provided
+		if config.FFTPointsT != nil && config.SRSG1 != nil {


Should this be validated (right after entering NewIcicleDevice) for properties of SRSG1 and FFPointsT? Or they are validated at the caller?

jianoaix · 2024-11-20T19:51:17Z

encoding/icicle/device_setup.go

+	device := runtime.CreateDevice("CPU", 0)
+	if runtime.IsDeviceAvailable(&device) {
+		slog.Info("CPU device available, setting device")
+	}


Is it safe to assume the CreateDevice never fails for CPU?

I believe that's the case but not sure.

I'm not sure about the behavior.

It may be safer to return an error and bubble up

jianoaix · 2024-11-20T19:57:12Z

encoding/icicle/device_setup.go

+	}
+
+	slog.Info("CUDA device not available, falling back to CPU")
+	return setupCPUDevice()


The GPUEnable is instructed by the cmdline flag, should this not fallback? Or comment in the flag that it'll fallback if there is no GPU.

Added comment to flag

jianoaix · 2024-11-20T22:16:42Z

encoding/kzg/prover/multiframe_proof.go

@@ -13,66 +14,18 @@ import (
 	"github.com/consensys/gnark-crypto/ecc/bn254/fr"
 )

-type KzgCpuProofDevice struct {
+type KzgMultiProofGnarkBackend struct {


Shall we create a gnark folder similar to icicle to organize these implementations with gnark?

jianoaix · 2024-11-20T22:26:15Z

encoding/kzg/prover/icicle/multiframe_proof.go

+
+		sumVec, err := p.MsmBatchOnDevice(flattenStoreCopyToDevice, p.FlatFFTPointsT, int(numPoly)*int(dimE)*2)
+		if err != nil {
+			icicleErr = fmt.Errorf("msm error: %w", err)


here and below: why not return on such an error? The proof computing is already failed at these points

jianoaix · 2024-11-20T22:27:29Z

encoding/kzg/prover/noicicle.go

How is this used?

#895 (comment)

Essentially just makes it so we don't have to compile icicle libraries unless we explicitly add the build tag.

jianoaix · 2024-11-20T22:31:12Z

encoding/kzg/prover/prover.go

+
+func (p *Prover) createGnarkBackendProver(params encoding.EncodingParams, fs *fft.FFTSettings, ks *kzg.KZGSettings) (*ParametrizedProver, error) {
+	if p.Config.GPUEnable {
+		return nil, fmt.Errorf("GPU is not supported in default backend")


nit: "default" -> "gnark" (some occurrences below as well)

jianoaix · 2024-11-20T22:36:49Z

encoding/kzg/verifier/batch_commit_equivalence_test.go

@@ -14,17 +14,21 @@ import (
 )

 func TestBatchEquivalence(t *testing.T) {


It'll be nice to have a icicle-gnark equivalence test for proofs, but may need gpu setup for unit test... Maybe we can start with CPU (for both), in a follow up

jianoaix

Mostly lg. There were a couple comments before would be nice to address

jianoaix · 2024-11-28T01:05:13Z

encoding/icicle/device_setup.go

+	device := runtime.CreateDevice("CPU", 0)
+	if runtime.IsDeviceAvailable(&device) {
+		slog.Info("CPU device available, setting device")
+	}


It may be safer to return an error and bubble up

jianoaix · 2024-12-02T22:40:48Z

encoding/kzg/prover/icicle/multiframe_proof.go

+	preprocessDone := time.Now()
+
+	// Prepare data before GPU operations
+	flattenCoeffStoreFr := make([]fr.Element, len(coeffStore)*len(coeffStore[0]))


The computeCoeffStore may create a 1D result directly then this flattening can be saved

jianoaix · 2024-12-02T22:55:30Z

encoding/kzg/prover/proof_backend.go

+
+// Proof device represents a backend capable of computing KZG multiproofs.
+type KzgMultiProofsBackend interface {
+	ComputeMultiFrameProof(blobFr []fr.Element, numChunks, chunkLen, numWorker uint64) ([]bn254.G1Affine, error)


So the blobFr are points on the blob's polynomial, but why doesn't this produce a single bn254.G1Affine (which is the proof)?

blobFr are the coefficients of the polynomial. We actually have some documentation about it here https://github.com/Layr-Labs/eigenda/blob/master/docs/spec/attestation/amortized-proving.md#amortized-multireveal-proof-generation-with-the-fft.

These two resources were good for understanding the implementation details (although I still don't fully grasp it yet):
https://alinush.github.io/2021/06/17/Feist-Khovratovich-technique-for-computing-KZG-proofs-fast.html
https://alinush.github.io/2020/03/19/multiplying-a-vector-by-a-toeplitz-matrix.html

jianoaix · 2024-12-02T23:02:01Z

encoding/rs/icicle/extend_poly.go

+// Encoding Reed Solomon using FFT
+func (g *RsIcicleBackend) ExtendPolyEval(coeffs []fr.Element) ([]fr.Element, error) {
+	// Lock the GPU for operations
+	g.GpuLock.Lock()


If it's not using GPU (which is possible for icicle), then wouldn't this lock hurt performance?

The perf is already quite poor on the CPU version of icicle. I would prefer to tackle that separately if that's okay.

…uteCoeffStore

jianoaix

LGTM, well done!

dmanc requested review from bxue-l2 and jianoaix November 14, 2024 08:31

samlaf reviewed Nov 14, 2024

View reviewed changes

api/clients/retrieval_client_test.go Outdated Show resolved Hide resolved

core/test/core_test.go Outdated Show resolved Hide resolved

disperser/cmd/encoder/flags/flags.go Outdated Show resolved Hide resolved

disperser/cmd/encoder/config.go Outdated Show resolved Hide resolved

samlaf reviewed Nov 14, 2024

View reviewed changes

jianoaix reviewed Nov 15, 2024

View reviewed changes

encoding/rs/serialization.go Outdated Show resolved Hide resolved

encoding/rs/noicicle.go Outdated Show resolved Hide resolved

encoding/rs/serialization.go Outdated Show resolved Hide resolved

dmanc commented Nov 15, 2024

View reviewed changes

jianoaix reviewed Nov 15, 2024

View reviewed changes

dmanc marked this pull request as ready for review November 16, 2024 00:03

dmanc force-pushed the gpu-encoder branch 3 times, most recently from d716e07 to 51b0403 Compare November 19, 2024 00:12

mooselumph self-requested a review November 19, 2024 19:14

bxue-l2 reviewed Nov 20, 2024

View reviewed changes

dmanc force-pushed the gpu-encoder branch from 910ca04 to 8529da3 Compare November 20, 2024 05:57

jianoaix reviewed Nov 20, 2024

View reviewed changes

dmanc force-pushed the gpu-encoder branch from 8caf7ea to 8825115 Compare November 26, 2024 01:27

dmanc and others added 10 commits November 27, 2024 00:33

GPU encoder init

d58f035

Update GPU code to icicle v3

7d90a7c

update

0cac49c

add logs for debugging

c66dc7a

Use RunDevice

65fdb61

Dockerfile fixes

d5ff7af

cleanup

0c3fb33

Cleanup / refactors

4946b5b

fix rebase issues

ef3df8f

Add RSEncode support to encoder server

5fac9d7

dmanc added 21 commits November 27, 2024 00:35

revert config changes on verifier

b6604ee

config for rs encoder

8bb640b

Refactor prover to use config

bec398b

Some pr feedback

01decb9

try this

5a61412

config

324f6b9

Try removing this

e806df4

add check

e9bf815

clearer fail msg

c294985

Add missing config

cf7cba5

Fix some issues with rebase

303050b

added comment on max_ntt_size

d3586df

Fix concurrency issue (i think) + respond to PR comments

5494c90

rm files

b2421e0

fix

15ad390

Fix

3235a3f

Add back lock

eaa093d

try gpulock + runondevice

f3ff995

Try this

6906988

Add panic handling

327d26f

works

851ed2b

dmanc force-pushed the gpu-encoder branch from 911c5ac to 851ed2b Compare November 27, 2024 00:38

dmanc added 2 commits November 27, 2024 00:44

remove unused errchan

914d94a

add back pprof

523980e

jianoaix reviewed Dec 2, 2024

View reviewed changes

Add error handling for CPU setup and work on flattened array for comp…

180cdf7

…uteCoeffStore

jianoaix approved these changes Dec 3, 2024

View reviewed changes

dmanc merged commit a84eff9 into Layr-Labs:master Dec 3, 2024
6 checks passed

hopeyen pushed a commit that referenced this pull request Dec 5, 2024

GPU accelerated encoder (#895)

b8c46a1

hopeyen pushed a commit that referenced this pull request Dec 6, 2024

GPU accelerated encoder (#895)

404480f

		@@ -14,17 +14,21 @@ import (
		)

		func TestBatchEquivalence(t *testing.T) {

GPU accelerated encoder #895

GPU accelerated encoder #895

Conversation

dmanc commented Nov 14, 2024 • edited Loading

Why are these changes needed?

Checks

samlaf left a comment • edited Loading

Choose a reason for hiding this comment

samlaf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jianoaix left a comment

Choose a reason for hiding this comment

dmanc commented Nov 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmanc Nov 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmanc commented Nov 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jianoaix left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jianoaix left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmanc commented Nov 14, 2024 •

edited

Loading

samlaf left a comment •

edited

Loading

dmanc Nov 16, 2024 •

edited

Loading